Penalized estimation for Big Data in Regression Problems and its Geometry

Abstract: I will present recent results obtained in [1] and [2] jointly with M. Bogdan, X. Dupuis, B. Kolodziejek, U. Schneider, T. Skalski, P. Tardivel and M. Wilczynski.

It is well known that LASSO discovers zero coefficients of the vector $b$ in the regression equation $Y=Xb+\varepsilon$ where $X$ is the data matrix and $Y$ the response vector. In fact LASSO estimates the sign of the coefficient vector $b$ ($ b_i$'s positive, negative or null). The sign is called the model(pattern) of LASSO. In the LASSO estimator the $\ell^1$ penalty is employed.

In the study of Big Data one needs to identify more informative patterns of the vector $b$. These leads to use penalties different from the $\ell^1$ penalty and to get more dimensionality reduction.

We define the pattern of any estimator with polyhedral penalty, i.e. the unit ball $B$ with respect to the penalty norm is a convex polyhedron. Surprising links between the pattern of a penalized estimator and the geometry of the convex polytope $B^*$ will be explained.

We study in detail estimation with a sorted $\ell^1$ penalty, called SLOPE. Its dual ball $B^*$ is a signed permutahedron. SLOPE is a popular method for dimensionality reduction in the high-dimensional regression, encompassing the LASSO estimator but also the $l^\infty$ penality. Indeed, some coefficients of the estimator $\hat b ^{\rm SLOPE}$ are null (sparsity) and others are equal in absolute value (clustering). Consequently, irrelevant predictors are eliminated and groups of predictors having the same influence on the response vector are identified. The SLOPE pattern of a vector $b$ provides: the sign of its components, clusters (components equal in absolute value) and clusters ranking.

In our research we give an analytical necessary and sufficient condition for SLOPE pattern recovery of an unknown vector $b$ of regression coefficients. Such condition is called Irrepresentability(IR) condition. For any polyhedral penalty we find a geometric IR condition.

[1] P. Graczyk, U. Schneider, T. Skalski, P. Tardivel, A Unified Framework for Pattern Recovery in Penalized and Thresholded Estimation and its Geometry, Journal of Optimization Theory and Applications(2026) vol. 208(1), 1-41.

[2] M. Bogdan, X. Dupuis, P. Graczyk, B. Kolodziejek, T. Skalski, P. Tardivel, M. Wilczynski, Pattern recovery by SLOPE, Applied and Computational Harmonic Analysis 80(2026), 1-25.

machine learningprobabilitystatistics theory

Audience: researchers in the discipline

---

Gothenburg statistics seminar

Series comments: Gothenburg statistics seminar is open to the interested public, everybody is welcome. It usually takes place in MVL14 (http://maps.chalmers.se/#05137ad7-4d34-45e2-9d14-7f970517e2b60, see specific talk). Speakers are asked to prepare material for 35 minutes excluding questions from the audience.

---